Methods and apparatus for fluorescence lifetime imaging with pulsed light

ABSTRACT

A light source may illuminate a scene with pulsed light that is pulsed non-periodically. The scene may include fluorescent material that fluoresces in response to the pulsed light. The pulsed light signal may comprise a maximum length sequence or Gold sequence. A lock-in time-of-flight sensor may take measurements of light returning from the scene. A computer may, for each pixel in the sensor, perform a Discrete Fourier Transform on measurements taken by the pixel, in order to calculate a vector of complex numbers for the pixel. Each complex number in the vector may encode phase and amplitude of incident light at the pixel and may correspond to measurements taken at a given time interval during the pulsed light signal. A computer may, based on phase of the complex numbers for a pixel, calculate fluorescence lifetime and scene depth of a scene point that corresponds to the pixel.

RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/323,621, filed Apr. 15, 2016 (the “Provisional Application”). Theentire disclosure of the Provisional Application is herein incorporatedby reference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with Government support under Contract No.N00030-13-C-0005 awarded by the Department of the Navy. The Governmenthas certain rights in the invention.

FIELD OF TECHNOLOGY

The present invention relates generally to fluorescence lifetimeimaging.

COMPUTER PROGRAM LISTING

Attached are three computer program files, each created as a .txt fileon Feb. 22, 2017: (1) OpticaDataProcess.txt with a size of about 2 KB;(2) createFit.txt with a size of about 3 KB; and (3) FDToF_optica.txtwith a size of about 2 KB. These three computer program files comprisesource code for software employed in a prototype implementation of thisinvention.

BACKGROUND

Conventional fluorescence lifetime imaging (“FLI”) suffers from at leasttwo technical problems.

First, conventional FLI may require precise knowledge of the shape ofthe light waveform emitted by the FLI system to illuminate the scene. Itis challenging to obtain a precise knowledge of the shape of the lightwaveform emitted by the FLI system, because physical processes (e.g.,crosstalk, latency, electromagnetic interference, hysteresis, nonlinearresponses, or signal noise) in electronics or other circuitry (e.g., ina computer, driver, light source, or wiring) may distort the shape(e.g., phase or amplitude) of the emitted light signal, relative to theshape of the electric control signal that is generated by a timer tocontrol the lighting. For example, the electric control signal generatedto control the lighting may be a square wave, but the actual emittedlight waveform may be smoothed and curved. Such smoothing occurs in mostelectro-optical systems that tend to be low-pass systems that suppresshigh frequency components. In a conventional ToF system, in order toachieve precise knowledge of the shape of the light waveform that isemitted by the system, expensive equipment may be employed. Thisexpensive equipment may be limited to a first mode (e.g., pulsedillumination with non-periodic pulses) or limited to a second mode(e.g., periodically modulated illumination, such as sinusoidal or squarewave modulation), and thus (2) may be unable to function in both modes.As a result, conventional FLI cannot be performed by a general purpose,inexpensive consumer grade time-of-flight (“ToF”) camera.

Second, conventional FLI may require taking one or more calibrationmeasurements to determine the placement of the fluorescent samplerelative to the sensor. The distance or depth of the sample with respectto the sensor is an unknown that is typically calibrated. Typically,this calibration is repeated often, since a separate calibration isperformed for each different scene being imaged. This repetitive,time-consuming calibration is not desirable.

SUMMARY

In illustrative implementations of this invention, these two technicalproblems are solved. In illustrative implementations, the FLI systemaccurately measures fluorescence lifetime: (a) even though the preciseshape of the excitation light waveform is not known; and (b) evenwithout calibration at a known scene depth.

Put differently, in illustrative implementations, the FLI systemaccurately measures fluorescence lifetime even though the system maybe—loosely speaking —“blind” (i.e., the precise shape of the excitationlight waveform is not known) and “reference-free” (i.e., scene depth hasnot been previously determined by one or more separate calibrationmeasurements). Because the FLI imaging system can accurately measurefluorescence lifetime while operating “blind” and “reference-free”, thesystem may be implemented with an inexpensive, consumer-grade lock-inToF sensor and without spending time for distance calibration.

In illustrative implementations, a light source emitsamplitude-modulated light. The light emitted by the light sourceilluminates a scene and excites a fluorescent material in the scene. Inresponse to this excitation light, the fluorescent material emitsfluorescent light that travels to a lock-in ToF sensor. The lock-in ToFsensor measures incident light, including this fluorescent light. Insome cases, a bandpass filter (e.g., a bandpass dielectric filter) ispositioned in front of the lock-in ToF sensor and filters out excitationlight that reflects from the scene, such that the incident lightreaching the ToF sensor comprises fluorescent light (from thefluorescing material in the scene) and noise (such as ambient light).

In illustrative implementations, the FLI system determines, for eachpixel of the lock-in ToF sensor, respectively, the fluorescence lifetimeand scene depth for a point in the scene that corresponds to the pixel.

In illustrative implementations: (A) the amplitude-modulated lightwaveform is periodic (such as sinusoidal wave or a square wave); and (B)a computer calculates, for each pixel of the lock-in ToF sensor,respectively, the fluorescence lifetime and scene depth for a point inthe scene that corresponds to the pixel, by computing a closed formsolution or by computing a non-linear least squares solution or othernon-linear inverse solution.

Alternatively, in some implementations: (A) the amplitude-modulatedlight waveform comprises a binary coded pulsed signal, where the pulsesare not periodic (such as a maximum length sequence or Gold sequence);and (B) a computer calculates, for each pixel of the lock-in ToF sensor,respectively, the fluorescence lifetime and scene depth for a point inthe scene that corresponds to the pixel, by computing a non-linear leastsquares solution.

In some implementations, the FLI system is configured: (A) to operate ineither a periodic illumination mode (in which amplitude modulation ofthe excitation waveform is periodic, such as a sinusoidal or squarewave) or a pulsed illumination mode (in which the excitation waveform isa binary coded, pulsed signal, in which the pulses are not periodic);and (B) to switch between these two modes by, among other things (i)changing the control signal that controls amplitude modulation of lightemitted by the light source, and (ii) changing an algorithm used toprocess the measurements taken by the ToF sensor

This invention has many practical applications, including (a)microscopy; and (b) large-scale fluorescence lifetime estimation, suchas for detection of counterfeit banknotes.

The Summary and Abstract sections hereof: (a) do not limit thisinvention; (b) are intended only to give a general introduction to someillustrative implementations of this invention; (c) do not describe allof the details of this invention; and (d) merely describe non-limitingexamples of this invention. This invention may be implemented in manyother ways. Likewise, the description of this invention in the Field ofTechnology section is not limiting; instead it identifies, in a general,non-exclusive manner, a field of technology to which someimplementations of this invention generally relate.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows hardware for an FLI system.

FIG. 2, FIG. 3, and FIG. 4 are flowcharts for FLI methods that employ aperiodic waveform of amplitude-modulated light. In FIG. 2, depth andlifetime are obtained by either a closed-form solution or by anon-linear least squares solution. In FIG. 3, depth and lifetime areobtained by a non-linear least squares solution. In FIG. 4, depth andlifetime are obtained from a closed form solution.

FIG. 5 is a flowchart for an FLI method that employs pulsed light.

FIG. 6 shows an FLI method.

The above Figures show some illustrative implementations of thisinvention, or provide information that relates to those implementations.The examples shown in the above Figures do not limit this invention.This invention may be implemented in many other ways.

DETAILED DESCRIPTION

Illustrative Hardware

FIG. 1 shows hardware for an FLI system 100, in an exemplaryimplementation of this invention. In FIG. 1, an illumination system 107illuminates a s scene 151 that includes a fluorescent material 150. AnFPGA (field-programmable gate array) integrated circuit 103 may generatecontrol signals that are communicated, via a PLL (phase-locked-loop)101, to a time-of-flight sensor 105 and also to a driver 111 inillumination system 107. The driver 111 may in turn control a lightsource 115 and thus controls amplitude modulation of light emitted thelight source 115.

The amplitude-modulated light emitted by the light source 115 maycomprise a periodic, continuous-wave signal (such as a sine wave or asquare wave) or may comprise a coded, binary, pulsed light signal inwhich the pulses are not periodic (such as a maximum length sequence ora Gold sequence). The light source 115 may comprise, for example, one ormore LEDs (light-emitting diodes), lasers, electronic white lightsources, or supercontinuum light sources. For example, the light source115 may emit coherent light, incoherent light, collimated light oruncollimated light. The spectrum of light emitted by the light sourcemay peak in a fluorescence absorption band of the fluorescent material150.

Light that travels from the scene 151 in the direction of ToF sensor 105may include both (1) reflected excitation light (i.e., light from thelight source 115 that reflects from the scene) and (2) fluorescent lightemitted by the fluorescent material 150, due to fluorescence that occursin response to light from light source 115 striking the fluorescentmaterial 150. A filter 121 may filter out reflected excitation light.For example, the filter 121 may comprise a dielectric filter. The filter121 may comprise a bandpass, lowpass or hi-pass filter.

Optics 123 may focus light onto a sensor pixel array of the ToF sensor105. For example, optics 123 may comprise one or more lens, mirrors orother refractive or reflective optical elements.

The ToF sensor 105 may comprise a lock-in ToF sensor, which includes anarray of lock-in pixels. For example, the pixels in the ToF sensor 105may comprise: (a) CMOS-CCD lock-in pixels, or (b) PMD (photonic mixerdevice) pixels. Also, for example, each pixel in the ToF sensor maycomprise a so-called “1-tap” pixel, a so-called “2-tap” pixel, aso-called “4-tap” pixel, or a so-called “n-tap” pixel. Also, forexample, the ToF sensor 105 may comprise a time-of-flight sensordescribed in a Listed Patent (as that term in defined herein) or mayinclude sensor pixels described in a Listed Patent. The pixels of theToF sensor 105 may output analog signals that encode measurements takenby the ToF sensor 105.

Alternatively, the pixels of the ToF sensor 105 may output digitalsignals that encode measurements taken by the ToF sensor 105. Forexample, in some cases, each pixel of the ToF sensor 105 performssingle-photon synchronous detection (SPSD). SPSD involves measurementswith single-photon avalanche diodes (SPADs).

The lock-in ToF sensor 105 may take measurements of incident light andmay output analog signals that encode these measurements. An ADC(analog-to-digital-converter) 131 may convert these analog signals intodigital signals. The FPGA may receive this digitized measurement dataand may send this data to a computer 133. The computer 133 may performan algorithm that recovers, for each pixel in the lock-in ToF sensor,the fluorescence lifetime and distance of a scene point that correspondsto the pixel. The computer 133 may store and retrieve data from memorydevice 135. The computer 133 may interface with human user(s) via one ormore input/output devices 137, such as a keyboard, mouse, displayscreen, interactive touch display screen, microphone, speaker, or haptictransceiver.

This invention is not limited to the hardware and hardware configurationshown in FIG. 1. This invention may be implemented in many other ways.

In illustrative implementations, the FLI system may be configured forhomodyne detection, in the sense that the ToF sensor may compare (e.g.,cross-correlate) an electric reference signal and a light signalincident on the ToF sensor, where each of these signals is ultimatelyderived from the same electronic timer signal.

Periodic Illumination Mode

As noted above, in some implementations, the FLI system (e.g., 100)operates in a periodic illumination mode. While operating in periodicillumination mode, the light emitted by the light source (e.g., 115) maybe an amplitude-modulated periodic signal, such as a sine wave, squarewave or other AMCW (amplitude-modulated continuous wave) signal.

For example, in some implementations, while operating in periodicillumination mode, the FLI system (e.g., 100) may probe the scene byilluminating the scene with a light signal p(t), the amplitude of whichis periodically modulated. For example, the amplitude-modulated lightsignal p(t) may be a sinusoidal signal (e.g., an AMCW light signal) offormp(t)=1+p ₀ cos(ωt)  Equation 1where p₀ is amplitude of the modulated light signal and ω is angularfrequency of the amplitude modulation. More generally, while operatingin periodic illumination mode, the amplitude-modulated light signal p(t)may be any periodic signal p(t)=p(t+T), where T is the time period orrepetition period. For example, p(t) may comprise a square wave or atriangular wave.

The reflected signal may be given byr(t)=(p*h _(SI))=∥ĥ(0)|+|ĥ(ω)|p ₀ cos(ωt+∠ĥ(ω))  Equation 2where (1) t is time, (2) h_(SI)=h(t) is the time-dependent,shift-invariant scene response function, and (3) ĥ(ω) is the FourierTransform of h(t).

In some implementations, when imaging fluorescence from a scene, h(t) isthe fluorescent response at depth d meters using the time-dependentfunction,

${h_{SI}(t)} = {{h(t)} = {{{\rho\delta}( {t - t_{0}} )} + {\mu\; e^{- \frac{t - t_{0}}{\tau}}{\Pi( {t - t_{0}} )}}}}$where (1) t₀=2d/c is the round trip, time delay due to an object or asample at distance d from the sensor, (2) c is the speed of light, (3) ρis albedo, (4), μ is luminance of fluorescence, (5) τ is the fluorescentlifetime and (6) Π(t) is the Heaviside function. Alternatively, μ may beany other measure of intensity of fluorescent light.

The reflected signal may be cross-correlated by a lock-in ToF sensor(e.g., 105), resulting in measurements,

$\begin{matrix}{{m(t)} = {{{\hat{h}(0)}} + {{{\hat{h}(\omega)}}\frac{p_{0}^{2}}{2}{\cos( {{\omega\; t} - {\angle\;{\hat{h}(\omega)}}} )}}}} & {{Equation}\mspace{14mu} 3}\end{matrix}$

Next, the lock-in ToF sensor may estimate the amplitude and depth/phaseinformation based on a so-called “four bucket” method. In thefour-bucket method, the lock-in ToF sensor may record four equidistantmeasurements m(πk/2ω), k=0, 1, 2, 3. A computer may use thesemeasurements to compute intermediate values

${M_{\omega}^{0,2} = {{{m(0)} - {{m( \frac{\pi}{\omega} )}\mspace{14mu}{and}\mspace{14mu} M_{\omega}^{3,1}}} = {{m( \frac{3\pi}{2\omega} )} - {m( \frac{\pi}{2\omega} )}}}},$and may in turn use these intermediate values to calculate a complexnumber z_(ω), wherez _(ω) =M _(ω) ^(0,2) +jM _(ω) ^(3,1).  Equation 4

In some implementations of this invention, the FLI system may operate inperiodic illumination mode and a complex number z_(ω) may be calculatedin accordance with Equation 4 for each pixel of the lock-in ToF camera,for each frequency ω of amplitude modulation in the sweep. For a givenmodulation frequency ω=ω₀, amplitude and phase estimates may becalculated according to

$\begin{matrix}{\underset{\underset{{Amplitude}\mspace{14mu}{Estimate}}{︸}}{{{\hat{h}( \omega_{0} )}} \approx {{{z( \omega_{0} )}}/p_{0}^{2}}}\mspace{14mu}{and}\mspace{14mu}\underset{\underset{{Phase}\mspace{14mu}{Estimate}}{︸}}{{{\angle\;\hat{h}( \omega_{0} )} \approx {\angle\;{z( \omega_{0} )}}},}} & {{Equation}\mspace{14mu} 5}\end{matrix}$respectively.

In Equation 5, the estimation of amplitude |ĥ(ω₀)| depends on p₀, where(as noted above) p₀ is amplitude of the modulated light signal. However,in Equation 5 phase may be calculated by ∠ĥ(ω₀)≈∠z_(ω) ₀ , withoutprecise knowledge of the shape of light signal p₀ (except forfrequency). This is advantageous, because the unknowns that we seek torecover in some implementations, namely, lifetime and depth parameters,may be extracted from phase without having to estimate amplitude|ĥ(ω₀)|, as discussed in more detail below.

As noted above, conventional FLI systems may require both (a) preciseknowledge of the illumination waveform and (b) repeated calibrations fordepth. In contrast, in some implementations of this invention, the FLIsystem, may be loosely said to be operating “blind”, in that the FLIsystem may accurately estimate depth and fluorescence lifetime,regardless of whether the exact shape of the illumination waveformemitted by the system's active light source (e.g., 115) is known inadvance. Specifically: In some implementations of this invention, whenthe FLI system is operating in in periodic illumination mode: (A) thefrequency of the amplitude modulation of the illumination light signalwaveform is known (because the FPGA and PLL are controlling thefrequency), and (b) phase of the amplitude modulation of theillumination light signal is not known in advance from separatecalibration measurements, but is instead extracted from the same set ofToF sensor measurements as the fluorescence lifetime. (Specifically, insome implementations, the phase is extracted from a set of ToF sensormeasurements, and fluorescence lifetime and scene depth are calculatedbased on the phase, and thus the phase, lifetime and scene depth areextracted from the same set of ToF sensor measurements). This is incontrast to conventional FLI, in which phase of the amplitude modulationmay be known in advance by separate calibration measurements, and thisknown phase may be used to help extract fluorescence lifetime from alater, separate set of ToF sensor measurements.

Furthermore, in some implementations of this invention, when the FLIsystem is operating in periodic illumination mode, the amplitude of themodulation of the illumination light signal is not known in advance fromseparate calibration measurements, but instead the phase, lifetime andamplitude may be extracted from the same set of ToF sensor measurements.This is in contrast to conventional FLI, in which the precise amplitudeof the modulation may be known in advance.

In some implementations of this invention, while the FLI system isoperating in periodic illumination mode, measurements at multiplefrequencies (of amplitude modulation) may be used to estimate depth dand fluorescence lifetime τ.

For example, in some implementations of this invention: (a) thefrequency of amplitude modulation of the light is swept; and (b) foreach pixel in the lock-in ToF sensor, a computer computes a vector z,where each element of the vector z respectively, is a complex numberthat (i) is computed based on measurements of incident light taken bythe pixel at a particular frequency of amplitude modulation, and (ii)encodes phase and amplitude of light incident on the pixel.

In some implementations, a computer may, for each pixel in the ToFsensor: (1) compute the phase of vector z to produce a vector ≮z, suchthat each element of vector ≮z is the phase of the corresponding elementof vector z; and (2) based on vector ≮z, compute fluorescence lifetimeand depth of a scene point that corresponds to the pixel.

Each element in the vector z may correspond to a different frequency ina sweep. For example, in some implementations when the FLI system isoperating in periodic illumination mode: (1) the frequency of amplitudemodulation of the light may be swept; and (2) each complex value z_(ω),respectively, in the vector z is computed based on measurements taken ata frequency ω.

For example, in some implementations, the frequency of amplitudemodulation of the light is swept, and, for each pixel in the lock-in ToFsensor: (1) a vector of measurements m_(ω) is acquired for eachfrequency ω of amplitude modulation in the sweep; (2) a computercalculates a vector z, where each element z_(ω), respectively, in vectorz is a complex number that encodes phase and amplitude of light incidenton the pixel and that is computed based on a vector of measurementsm_(ω) acquired at a frequency ω; (4) a computer calculates the phase ofvector z to produce a vector ≮z, such that each element of vector ≮z isthe phase of the corresponding element of vector z; and (5) based onvector ≮z, a computer calculates fluorescence lifetime and depth of ascene point that corresponds to the pixel.

The number of measurements taken at a given frequency (of amplitudemodulation) by a pixel may vary, depending on the particularimplementation of this invention. For example, in some implementations:(a) a “four bucket” calculation is used in an interim step; (b) eachpixel takes four measurements for each frequency ω of amplitudemodulation, and (c) the vector of measurements m_(ω) for a given pixelfor a given frequency ω consists of four measurements and is

$m_{\omega} = {\lbrack {{m(0)}\mspace{14mu}{m( \frac{\pi}{2\omega} )}\mspace{14mu}{m( \frac{\pi}{\omega} )}\mspace{14mu} m\mspace{11mu}( \frac{3\pi}{2\omega} )} \rbrack.}$Likewise, in some implementations: (a) each pixel takes two measurementsfor a given frequency ω of amplitude modulation, and thus (b) the vectorof measurements m_(ω) for a given pixel for a given frequency ω consistsof two measurements. More generally, in some implementations: (a) eachpixel takes Y measurements for a given frequency ω of amplitudemodulation; and thus (b) the vector of measurements m_(ω) for a givenfrequency ω for a given pixel consists of Y measurements.

In some implementations, the frequency of amplitude modulation is sweptin equal increments of frequency. For example, in some implementations:(1) the frequency of amplitude modulation is swept in equal incrementsof frequency, such that the frequencies in the sweep are ω=nω₀, n=a, . .. , N+a−1, where N is the number of frequencies in the sweep, a is aninteger greater than or equal to zero, and ω₀ is a frequency ofamplitude modulation; and (2) for each pixel of the ToF sensor,respectively (i) a computer calculates a vector z of complex numbers,(ii) each element of vector z corresponds to a different frequency inthe sweep, (iii) z=|z_(aω) ₀ z_((a+1)ω) ₀ z_((a+2)ω) ₀ . . .z_((N+a−1)ω) ₀ |, (iv) each complex number z_(ω), respectively, in thevector z encodes phase and amplitude of light incident on the pixel andis computed based on a vector of measurements m_(ω) taken by the pixelat a frequency ω, (v) a computer calculates the phase of vector z toproduce a vector ≮z, such that each element of vector ≮z is the phase ofthe corresponding element of vector z; and (vi) based on vector ≮z, acomputer calculates fluorescence lifetime and depth of a scene pointthat corresponds to the pixel.

In some implementations, a “four bucket” calculation is employed in aninterim step. For example, in some implementations, the frequency ofamplitude modulation is swept and, for each given pixel in the ToFsensor: (1) the vector of measurements m_(ω) for a given frequency ωconsists of four measurements and is

${m_{\omega} = \lbrack {{m(0)}\mspace{14mu}{m( \frac{\pi}{2\omega} )}\mspace{14mu}{m( \frac{\pi}{\omega} )}\mspace{14mu} m\mspace{11mu}( \frac{3\pi}{2\omega} )} \rbrack};$(2) each complex value z_(ω), respectively, in the vector z is computed,based on a vector of measurements m_(ω) taken at a frequency ω; (3) eachcomplex value z_(ω), respectively, in the vector z is estimated by a“four bucket” method according to Equation 4; (4) each complex valuez_(w), respectively, in the vector z encodes phase and amplitude oflight incident on the pixel; (5) a computer calculates the phase ofvector z to produce a vector ≮z, such that each element of vector ≮z isthe phase of the corresponding element of vector z and (6) based onvector ≮z, a computer calculates fluorescence lifetime and amplitude oflight incident on the pixel. Alternatively, in step (3) of the precedingsentence, a computer may perform one or more other conventional ToFcalculations (instead of a “four bucket” calculation according toEquation 4) to estimate (based on the vector of complex values m_(ω)taken at frequency ω) each complex value z_(ω), respectively, in thevector z.

In some implementations of this invention: (a) an FLI system operates inperiodic illumination mode; (b) the frequency of amplitude modulation isswept in equal increments such that, in each increment of the sweep,frequency changes by ω₀, and (c) for each pixel of the ToF sensor, acomputer calculates scene depth d and fluorescence lifetime τ (of ascene point that corresponds to the pixel) by solving a non-linearinverse problem.

For example, in some implementations, while the FLI system is operatingin periodic illumination mode, a computer may (for each pixel of the ToFsensor) calculate scene depth d and fluorescence lifetime τ (of a scenepoint that corresponds to the pixel) by solving a non-linear leastsquares problem according to

$\begin{matrix}{\underset{\{{d,\tau}\}}{\arg\mspace{11mu}\min}{\sum\limits_{n = a}^{N + a - 1}{{{z_{n\;\omega_{0}}} + {2( \frac{d}{c} )n\;\omega_{0}} + {\tan^{- 1}( {n\;\omega_{0}\tau} )}}}^{2}}} & {{Equation}\mspace{14mu} 6}\end{matrix}$where (1) N is the number of frequencies of amplitude modulation in thesweep; (2) a is an integer that is greater than or equal to zero; (3) ω₀is a frequency of amplitude modulation, (4) each z_(nω) ₀ is a complexnumber that is computed based on measurements of incident light by thepixel at frequency nω₀ and that encodes phase and amplitude of lightincident on the pixel; and (5) ≮z_(nω) ₀ is the phase of z_(nω) ₀ . Forexample, there may be a separate complex number z_(nω) ₀ for eachdifferent frequency of amplitude modulation in the sweep. In some cases,a computer may perform a conventional “four bucket” calculation (e.g.,according to Equation 4) to calculate each complex number z_(nω) ₀ inEquation 6. In some other cases, a computer may perform anotherconventional ToF calculation (which is not a “four bucket” method and isnot according to Equation 4) to calculate each z_(nω) ₀ in Equation 6.

In some cases, when performing calculations according to Equation 6: (1)a computer may calculate a vector z that consists of the elements z_(nω)₀ n=a, . . . , N+a−1; and (2) a computer may calculate a vector ≮z bytaking the phase of vector z, such that each element of vector ≮z is thephase of the corresponding element of vector z (e.g., such that, for anygiven n, ≮z_(nω) ₀ in vector ≮z is the phase of z_(nω) ₀ in vector z).

The sigma notation in Equation 6 may represent a sum of terms, whereeach term is computed for a different frequency of amplitude modulationin a sweep. For example, in Equation 6, if a=0, and ω₀/2π is 1 MHz, andN=41, then the sweep of frequency is from 0 MHz to 40 MHz in incrementsof 1 MHz. Likewise, in Equation 6, if a=10, and w₀/2π is 1 MHz, andN=41, then the sweep of frequency is from 10 MHz to 50 MHz in incrementsof 1 MHz.

In some implementations, in order to solve the non-linear least squaresproblem in Equation 6 or in Equation 13, a computer may perform atrust-region-based algorithm with least absolute residual criterion. Forexample, the trust-region-based algorithm may be as described in AnInterior Trust Region Approach for Nonlinear Minimization Subject toBounds, by T. Coleman et al., published in Society for Industrial andApplied Mathematics Journal, Vol. 6, No. 2, pp. 418-445, May 1996 (the“Coleman Paper”). However, this invention is not limited to atrust-region-based algorithm; other least square algorithms may beemployed. Furthermore, this invention is not limited to least squaresalgorithms. For example, a computer may perform other fitting algorithmsto solve for lifetime τ and depth d, or may calculate a closed formsolution for lifetime τ and depth d.

Equation 6 assumes that the frequency of amplitude modulation is sweptusing equal intervals of frequency (e.g., frequency may be swept inincrements of 1 MHz, for each discrete step in the sweep).Alternatively, this invention may be implemented, such that frequency isswept, where the increments in frequency are not constant (e.g., thefrequency might increase by 1 MHz from the first to the second frequencyof the sweep, and may increase by 2 MHz from the second to the thirdfrequency in the sweep). In these alternative implementations (involvinga sweep with non-equal increments of frequency), for each pixel of theToF sensor, a computer may calculate scene depth d and fluorescencelifetime τ (of a scene point that corresponds to the pixel) by solvingan optimization problem.

In an experiment, the accuracy of a prototype of this inventionoperating in periodic illumination mode was evaluated as follows: Inthis experiment, frequency of amplitude modulation of light was sweptfrom 1 MHz to 40 MHz in increments of 1 MHz—that is, ω₀/2π was 1 MHz andToF measurements were taken at equidistant intervals of frequency ofamplitude modulation, such that the frequencies in the sweep were nω₀,n=1, 2, . . . , 40. In this experiment, the ToF sensor took measurementsat each frequency in the sweep. In this experiment, for each of tenpixels in the lock-in ToF sensor, the FLI system estimated fluorescencelifetime τ and distance d. In this experiment: (a) the average estimatedlifetime τ and average estimated distance {tilde over (d)}, as estimatedby the FLI system for the ten pixels, were 31.024 ns and 2.4961 m,respectively; and (b) the actual lifetime τ and actual distance d were32 ns and 2.5 m, respectively. In this experiment, for these ten pixels:(a) the log scale root mean square error for the estimated lifetime{tilde over (τ)} ranged from −8.66 to −9.85; and (b) the signal-to-noiseratio ranged from 43.10 dB to 46.00 dB. Neither the prototype nor thisexperiment limit this invention. This invention is not limited to theprototype, it may be implemented in many other ways. This invention isnot limited to the use scenario and technique used in this experiment,it may be implemented in many other ways.

FIG. 2 is a flowchart for an FLI method that employs periodicamplitude-modulated light, in an illustrative implementation of thisinvention. In the example shown in FIG. 2, the method includes thefollowing steps: A computer may cause an electrical reference signal tobe sent to a light source and to a ToF sensor. The reference signalcontrols amplitude modulation of light emitted by the light source (Step201). The light source illuminates a fluorescent sample in a scene. Theamplitude modulation of light emitted by the light source comprises aperiodic, continuous wave, such as a sine wave or square wave. Thefrequency of the periodic wave may be swept in a frequency sweep thatconsists of N different amplitude modulation frequencies (Step 202).Light incident on a bandpass filter may include both reflectedexcitation light and fluorescent light, where: (a) the reflectedexcitation light comes from the light source and reflects from thescene; and (b) the fluorescent light is emitted by the fluorescentsample in response to light from the light source. The bandpass filtermay filter out reflected excitation light. The light that passes throughthe bandpass filter may then travel to the ToF sensor (Step 203). Eachpixel of the ToF sensor may measure cross-correlation of the referencesignal and of light incident on the pixel. For each pixel in the ToFsensor, a first vector of N complex values may be computed for thefrequency sweep, such that each complex value in the first vectorcorresponds to measurements by the pixel at a given frequency in thefrequency sweep and encodes phase and amplitude of light incident at thepixel. For example, each of these complex values may be computed by the“four bucket” method and may comprise a z_(ω) as described in Equation 4(Step 204). For each pixel in the ToF sensor, a computer may computephase of these complex values and may output a second vector that has Nelements, such that each element of the second vector is the phase of acorresponding complex value in the first vector (Step 205). For eachpixel in the ToF sensor, a computer may, based on this second vector ofphases, solve for distance d and fluorescence lifetime τ of a scenepoint that corresponds to the pixel. For example, a computer may, basedon this second vector of phases, calculate: (a) a closed form solutionfor d and τ, respectively; or (b) a non-linear least squares solutionfor d and τ, respectively (Step 206).

FIG. 3 is another flowchart for an FLI method that employs periodicamplitude-modulated light, in an illustrative implementation of thisinvention. In the example shown in FIG. 3, the method includes thefollowing steps: The amplitude modulation of light emitted by the lightsource may comprise a periodic, continuous wave, such as a sine wave orsquare wave. The frequency of the periodic wave may be swept in afrequency sweep that consists of N different amplitude modulationfrequencies (Step 301). For each pixel in the ToF sensor, a vector z ofN complex values may be computed for a frequency sweep, such that eachcomplex value in the vector z corresponds to measurements by the pixelat a given frequency in the frequency sweep and encodes phase andamplitude of light incident at the pixel. For example, each of thesecomplex values may be computed by the “four bucket” method and maycomprise a z_(ω) as described in Equation 4 (Step 302). For each pixelin the ToF sensor, a computer may calculate (based on vector z) anon-linear least squares solution for distance d and fluorescencelifetime τ, respectively, of a scene point that corresponds to the givenpixel. For example, the non-linear least squares solution for d and τmay be calculated according to Equation 6. A computer may employ Matlab®inbuilt fitting functions to solve this non-linear least squares problem(Step 303).

FIG. 4 is another flowchart of an FLI method that employs periodicamplitude-modulated light, in an illustrative implementation of thisinvention. In the example shown in FIG. 4, the method includes thefollowing steps: The light emitted by the light source may comprise aperiodic, amplitude-modulated continuous wave, such as a sinusoidal waveor square wave. The frequency of the periodic wave may be swept in afrequency sweep that consists of N different amplitude modulationfrequencies (Step 401). For each pixel in the ToF sensor, a vector z ofN complex values may be computed for a frequency sweep, such that eachcomplex value in the vector z corresponds to measurements by the pixelat a given frequency in the frequency sweep and encodes phase andamplitude of light incident at the pixel. For example, each of thesecomplex values may be computed by the “four bucket” method and maycomprise a z_(ω) as described in Equation 4 (Step 402). In some cases:(a) for each pixel in the lock-in-ToF sensor, a computer may calculate(based on vector z) a closed form solution for distance d andfluorescence lifetime τ, respectively of a scene point that correspondsto the given pixel; and (b) the closed form solution for d and τ may becalculated according to the linear system of equationsz _(n+1)(1+jω _(n+1)τ)=z _(n)(1+jω _(n)τ)e ^(−jω) ⁰ ^(2d/c)  Equation 7where (1) the sweep comprises amplitude modulation frequencies ω_(n),(2) ω_(n)=nω₀, (3) ω₀ is a frequency of amplitude modulation, (4) n=a,a+1, a+2, . . . , N+a−1, (5) “a” is an integer that is greater than orequal to zero, (6) C is speed of light, and (7) each z_(n) and z_(n+1)is an element of the vector z (Step 403). For example, the linear systemof equations in Equation 7 may be solved with any four contiguousinteger values of n, such that l≤n<l+4 where l is an integer.

For example, in Step 403, if n=2, 3, 4, . . . 40, and ω⁰/2π is 1 MHz,then the sweep of frequency of amplitude modulation is from 2 MHz to 40MHz, in increments of 1 MHz.

In some implementations, the ToF sensor mentioned in FIG. 2, FIG. 3,FIG. 4, FIG. 5 or FIG. 6 may comprise ToF sensor 105 (in FIG. 1).

Pulsed Illumination Mode

As noted above, in some implementations, the FLI system (e.g., 100)operates in a pulsed illumination mode. While operating in pulsedillumination mode, the light emitted by the light source (e.g., 115) maybe an amplitude-modulated, coded, binary, pulsed, signal, where theamplitude modulation of the signal is not periodic. For example, whileoperating in pulsed illumination mode, the light emitted by the lightsource (e.g., 115) may comprise a maximum length sequence or a Goldsequence. For example, in some cases, the light signal emitted by thelight source comprises a 31-bit MLS (maximum length sequence) describedby the code MLS=0101110110001111100110100100001.

In some implementations, the FLI system operates in pulsed illuminationmode and measures fluorescence lifetime and scene depth—even withoutprecise knowledge of the shape of the pulsed light signal p(t) emittedby the system's light source (e.g., 115).

In some implementations, the measurements taken by the lock-in ToFsensor during pulsed illumination mode may be approximated as follows:

$\begin{matrix}{{m\mspace{11mu}(t)} \approx {\frac{1}{\Delta}{\sum\limits_{{n} \leq N_{0}}{{\hat{\phi}}_{n}{{{\hat{h}}_{n}}^{j\;\omega}}_{p^{nt}}}}}} & {{Equation}\mspace{14mu} 8}\end{matrix}$where (1) j is a unit imaginary number that is equal to a square root ofnegative one, (2) Δ is signal length, (2) ω_(p)=2π/Δ, (3) h_(SI)(t) isthe shift-invariant scene response; (4) ĥ(ω) is the Fourier transform ofh_(SI)(t), (5) ĥ_(n)=ĥ(ω_(p)n), and (6) ϕ approximates thecross-correlation function (i.e., the cross-correlation of the referenceelectric signal and the light signal incident on the ToF sensor) with2N₀+1 Fourier series coefficients, {circumflex over (ϕ)}_(n), wheren=−N₀, . . . , +N₀. As used herein, “signal length” means number ofsamples times the amount of time that it takes to acquire one sample. Asa non-limiting example: if the sample period is 1 ns and 100 samples aretaken, then “signal length” is 100 ns.

In some implementations:

$\begin{matrix}( {{Fourier}\mspace{14mu}{Transform}} ) & \; \\\begin{matrix}{{\hat{h}(\omega)} = {\int{{h(t)}e^{j\;\omega\; t}{dt}}}} \\{= {{\rho\; e^{{- j}\;{\omega{(\frac{2d}{c})}}}} + {\frac{\mu}{\frac{1}{\tau} + {j\;\omega}}e^{{- j}\;{\omega{(\frac{2d}{c})}}}}}}\end{matrix} & {{Equation}\mspace{14mu} 9}\end{matrix}$where (1) ω is angular frequency, (2) d is distance from scene (i.e.,scene depth), (3) τ is fluorescence lifetime, (4) c is speed of light,(5) e is the unique number whose natural logarithm is equal to one, andthus e≈2.71828, (6) μ is luminance of fluorescence, and (7) ρ is albedo.Alternatively, μ may be any other measure of intensity of fluorescentlight.

In some implementations, ĥ(ω) may be expressed in polar form asĥ(ω)=|ĥ(ω)|e^(j∠ĥ(ω)), where

$\begin{matrix}{{{\hat{h}(\omega)}} = \sqrt{\frac{( {\rho + {\mu\tau}} )^{2} + ({\omega\rho\tau})^{2}}{1 + ({\omega\tau})^{2}}}} & {{Equation}\mspace{14mu} 10} \\{{\angle\;{\hat{h}(\omega)}} = {{- {\tan^{- 1}( \frac{{\omega\mu\tau}^{2}}{\rho + {\mu\tau} + {\rho\mspace{11mu}({\omega\tau})^{2}}} )}} - {\frac{2d}{c}\omega}}} & {{Equation}\mspace{14mu} 11}\end{matrix}$

In some implementations: (1) light from the scene is filtered (e.g., bya dielectric, bandpass filter) to remove the reflected excitation light(i.e., to remove the amplitude-modulated light that was emitted by thelight source and that reflected from the scene), so that onlyfluorescent emission light from the fluorescent sample is recorded bythe ToF sensor; and thus, (2) ρ=0. Thus, in some implementations, byappropriately filtering the received light, Equation 11 simplifies to

${\angle\;{\hat{h}(\omega)}} = {{- {\tan^{- 1}({\omega\tau})}} - {\frac{2d\;\omega}{c}.}}$

Note that in Equation 11, the phase ∠ĥ(ω) of the spectrum encodes bothscene depth d and fluorescence lifetime τ.

In vector-matrix notation, the discretized system of equations inEquation 8 may be written asm=VD _({circumflex over (ϕ)}) ĥ  Equation 12where (1) m is K×1 vector of discretized ToF sensor measurementsm_(k)=m(kT_(s)), k∈[0, K−1], (2) T_(s) is sampling period, (3) V is aVandermonde matrix of size K×(2N₀+1) with matrix element[V]_(k,n)=e^(j(2π/Δ)T) ^(s) ^(nk), n∈[−N₀,+N₀], (4)D_({circumflex over (ϕ)}) is a (2N₀+1)×(2N₀+1) diagonal matrix withdiagonal entries [D]_(n,n)={circumflex over (ϕ)}_(n), and (5) ĥ is a(2N₀+1)×1 vector of the discretized spectrum (see Equation 9) withentries ĥ(2πn/Δ), n=−N₀, . . . , N₀.

Thus, in some implementations, the estimation problem is: given Kmeasurements m, estimate scene depth d and fluorescence lifetime τ.

In illustrative implementations, the emitted light signal and thereference signal applied to the lock-in ToF sensor may be the sameexcept for delays. The lock-in ToF sensor may take measurements thateffectively cross-correlate these two signals.

In some implementations of this invention, in which the FLI system isoperating in pulsed illumination mode: (1) the cross-correlation ϕ(t) isa real, time symmetric function; (2) thus the matrixD_({circumflex over (ϕ)}) does not contribute to the phase of vector ĥin Equation 12; (3) phase may, in turn, encode the scene depth andfluorescence lifetime parameters, see discussion of Equation 11; (4)thus the matrix D_({circumflex over (ϕ)}) (which does not contribute tophase in Equation 12) may be disregarded, when calculating the lifetimeand depth parameters from phase; and (5) cross-correlation functionϕ(t)—and the light waveform p(t) that is being cross-correlated with thereference signal—do not need to be known in order to calculate phaseaccording to Equation 12 and to extract depth and lifetime from phase.

In some implementations of this invention (in which the FLI system isoperating in pulsed illumination mode), the phase, lifetime and depthparameters are extracted from the same set of ToF sensor measurements.Specifically, in some implementations in which the FLI system isoperating in pulsed illumination mode: (a) phase is extracted from a setof ToF sensor measurements; (b) fluorescence lifetime and scene depthare calculated based on the phase; and (c) thus the phase, lifetime andscene depth are extracted from the same set of ToF sensor measurements.This is in contrast to conventional FLI, in which phase of the amplitudemodulation may be known in advance by separate calibration measurements,and this known phase may be used to help extract fluorescence lifetimefrom a later, separate set of ToF sensor measurements.

Furthermore, in some implementations of this invention (in which the FLIsystem is operating in pulsed illumination mode), the amplitude of themodulation of the illumination light signal is not known in advance fromseparate calibration measurements, but instead the phase, lifetime,depth and amplitude may be extracted from the same set of ToF sensormeasurements. This is in contrast to conventional FLI, in which theprecise amplitude of the modulation may be known in advance.

In some implementations, in which the FLI system is operating in pulsedillumination mode: (a) a computer may perform a Discrete FourierTransform (DFT) on the vector of measurements m for that pixel,resulting in vector g for that pixel, such that each element of vector gis a complex value that encodes both amplitude and phase of lightincident on that pixel, and (b) a computer may extract depth d andfluorescence lifetime τ from vector g by solving a non-linear leastsquares problem or other non-linear inverse problem. For example, acomputer may determine d and τ by solving a non-linear least squaresproblem according to:

$\begin{matrix}{\underset{\{{d,\tau}\}}{\arg\mspace{11mu}\min}{\sum\limits_{k = 0}^{K - 1}{{{g_{k}} + {2( \frac{d}{c} )k\;\omega_{p}} + {\tan^{- 1}( {k\;\omega_{p}\tau} )}}}^{2}}} & {{Equation}\mspace{14mu} 13}\end{matrix}$where (1) K is the number of elements in vector g, (2) ω_(p)=2π/Δ, (3) Δis signal length, (4) c is speed of light, (5) each element g_(k) of thevector g is a complex number that corresponds to measurement of incidentlight by the pixel at a different time during the pulsed signal, and (6)≮g_(k) is the phase of g_(k).

In some cases, when performing calculations according to Equation 13, acomputer may calculate a vector ≮g by taking the phase of vector g, suchthat each element of vector ≮g is the phase of the corresponding elementof vector g (e.g., for any given k, ≮g_(k) in vector ≮g is the phase ofg_(k) in vector g).

In some implementations, to solve the non-linear least squares problemin Equation 13, a computer may perform a trust-region-based algorithmwith least absolute residual criterion. For example, a computer mayperform a trust-region-based algorithm described in the Coleman Paper.

FIG. 5 is a flowchart for an FLI method that employs pulsed modulationof light, in an illustrative implementation of this invention. In theexample shown in FIG. 5, the method includes the following steps: Acomputer causes an electrical reference signal to be sent to a lightsource and to a ToF sensor. The reference signal controls amplitudemodulation of light emitted by the light source (Step 501). The lightsource illuminates a fluorescent sample in a scene. The amplitudemodulation of light emitted by the light source may comprise a binarypulsed waveform, such as a maximum length sequence or a Gold sequence(Step 502). Light incident on a bandpass filter may include bothreflected excitation light and fluorescent light, where: (a) thereflected excitation light comes from the light source and reflects fromthe scene; and (b) the fluorescent light is emitted by the fluorescentsample in response to light from the light source. The bandpass filtermay filter out reflected excitation light. The light that passes throughthe bandpass filter may then travel to the ToF sensor (Step 503). Eachpixel of the ToF sensor may measure cross-correlation of the referencesignal and of light incident on the pixel. For each pixel in the ToFsensor: (a) a vector m of N scalar values may be computed (e.g., suchthat each scalar value in the vector is a cross-correlation measured bythe pixel at a different time during the pulsed waveform); (b) acomputer may calculate a vector g by taking a Fourier transform of m(e.g., by multiplying by a discrete Fourier transform matrix); and (c) acomputer may calculate a non-linear inverse solution for distance d andfluorescence lifetime τ, respectively, of a scene point that correspondsto the given pixel. For example, a computer may calculate a non-linearleast squares solution for d and τ according to Equation 13. A computermay employ Matlab® inbuilt fitting functions to solve this non-linearleast squares problem (Step 504).

As noted above, conventional FLI devices suffer from at least twotechnological problems which may be solved by this invention. Inaddition, some conventional FLI devices suffer from a thirdtechnological problem: Specifically, for some conventional FLI devices,fluorescence lifetime must be known (e.g., from calibration) in order tosolve for scene depth. This invention may solve this third technologicalproblem also, because, in illustrative implementations of thisinvention, an FLI system may—without calibration to determinefluorescence lifetime—accurately measure both fluorescence lifetime anddepth.

Applications

In illustrative implementations of this invention, the FLI system hasmany practical applications. For example, in some implementations ofthis invention, the FLI system is employed for: (a) microscopy,including both wide-field microscopy and point-scanning microscopy; (b)biological imaging, including estimation of a fluorescence lifetime thatis less than one millisecond and more than one picosecond; (c)estimation of fluorescence lifetime for large-scale (macroscopic)scenes; and (d) detection of counterfeit banknotes, based on fluorescentcharacteristics of the banknotes.

In some implementations of this invention, the FLI system is well-suitedfor imaging at both a large scale and a small scale. This is because,among other things, the FLI system may determine depth and fluorescencelifetime on a per-pixel basis, regardless of the lateral scale of thescene being imaged.

FIG. 6 shows an FLI method, in an illustrative implementation of thisinvention. In the example shown in FIG. 6, the method includes thefollowing step: Based on measurements taken by a ToF sensor, a computermay, among other things: (a) calculate, for each given pixel in the ToFsensor, distance d and fluorescence lifetime r, respectively, of a scenepoint that corresponds to the given pixel; (b) output an amplitudeimage; (c) output a phase image; (d) output a depth map; (e) output afluorescence lifetime map; or (f) identify one or more materials (e.g.,fluorescent materials) in the scene (Step 601).

Prototype: Source Code

In the Computer Program Listing above, three computer program files arelisted. These three computer program files comprise software employed ina prototype implementation of this invention. To run these as Matlab®software files, the filename extension for these files would be changedfrom “.txt” to “.m”, “.mat”, and “.m”, respectively. Here is adescription of these three computer program files:

(1) OpticaDataProcess.txt: OpticaDataProcess is a function that: (a)takes, as an input, a vector of complex-valued measurements acquired bya lock-in ToF sensor; (b) unwraps phases of the complex-valuedmeasurements, (c) performs a non-linear fit by calling the createFitfunction, and (d) outputs fluorescence lifetime in nanoseconds anddistance (scene depth) in meters.

(2) createFit.txt: The createFit program performs a non-linear fit.

(3) FDToF_optica.txt: FDToF_optica is a function that: (a) takes, asinputs, depth and lifetime data outputted by the OpticaDataProcessfunction; (b) calculates other metrics (root-mean-square-error andsignal-to-noise ratio), and (c) outputs a table of the lifetime anddepth data and of the other metrics.

This invention is not limited to the software set forth in these threeprogram files. Other software may be employed. Depending on theparticular implementation, the software used in this invention may vary.

Computers

In illustrative implementations of this invention, one or more computers(e.g., servers, network hosts, client computers, integrated circuits,microcontrollers, controllers, field-programmable-gate arrays, personalcomputers, digital computers, driver circuits, or analog computers) areprogrammed or specially adapted to perform one or more of the followingtasks: (1) to control the operation of, or interface with, hardwarecomponents of an FLI system, including any ToF sensor, driver, lightsource, or ADC; (2) to perform a DFT; (3) to perform a “4-bucket”algorithm; (4) to calculate, based on ToF sensor measurements, a vectorof complex values that encode phase and amplitude of incident light; (5)to extract lifetime fluorescence and scene depth from phase; (6) toperform any other calculation, computation, program, algorithm, orcomputer function described or implied above; (7) to receive signalsindicative of human input; (8) to output signals for controllingtransducers for outputting information in human perceivable format; and(9) to process data, to perform computations, to execute any algorithmor software, and to control the read or write of data to and from memorydevices (items 1-9 of this sentence referred to herein as the “ComputerTasks”). The one or more computers (e.g. 103, 133) may be in anyposition or positions within or outside of the FLI system. For example,in some cases (a) at least one computer is housed in or together withother components of the FLI system, such as the ToF sensor orillumination system, and (b) at least one computer is remote from othercomponents of the FLI system. The one or more computers may communicatewith each other or with other devices either: (a) wirelessly, (b) bywired connection, (c) by fiber-optic link, or (d) by a combination ofwired, wireless or fiber optic links.

In exemplary implementations, one or more computers are programmed toperform any and all calculations, computations, programs, algorithms,computer functions and computer tasks described or implied above. Forexample, in some cases: (a) a machine-accessible medium has instructionsencoded thereon that specify steps in a software program; and (b) thecomputer accesses the instructions encoded on the machine-accessiblemedium, in order to determine steps to execute in the program. Inexemplary implementations, the machine-accessible medium may comprise atangible non-transitory medium. In some cases, the machine-accessiblemedium comprises (a) a memory unit or (b) an auxiliary memory storagedevice. For example, in some cases, a control unit in a computer fetchesthe instructions from memory.

In illustrative implementations, one or more computers execute programsaccording to instructions encoded in one or more tangible,non-transitory, computer-readable media. For example, in some cases,these instructions comprise instructions for a computer to perform anycalculation, computation, program, algorithm, or computer functiondescribed or implied above. For example, in some cases, instructionsencoded in a tangible, non-transitory, computer-accessible mediumcomprise instructions for a computer to perform the Computer Tasks.

Network Communication

In illustrative implementations of this invention, an electronic device(e.g., 103, 105, 111, 131, 135) is configured for wireless or wiredcommunication with other electronic devices in a network.

For example, in some cases, one or more components of the FLI system(e.g., 100) may each include a wireless communication module forwireless communication with other electronic devices in a network. Eachwireless communication module may include (a) one or more antennas, (b)one or more wireless transceivers, transmitters or receivers, and (c)signal processing circuitry. Each wireless communication module mayreceive and transmit data in accordance with one or more wirelessstandards.

In some cases, one or more of the following hardware components are usedfor network communication: a computer bus, a computer port, networkconnection, network interface device, host adapter, wireless module,wireless card, signal processor, modem, router, cables or wiring.

In some cases, one or more computers (e.g., 103, 133) are programmed forcommunication over a network. For example, in some cases, one or morecomputers are programmed for network communication: (a) in accordancewith the Internet Protocol Suite, or (b) in accordance with any otherindustry standard for communication, including any USB standard,ethernet standard (e.g., IEEE 802.3), token ring standard (e.g., IEEE802.5), wireless standard (including IEEE 802.11 (wi-fi), IEEE 802.15(bluetooth/zigbee), IEEE 802.16, IEEE 802.20 and including any mobilephone standard, including GSM (global system for mobile communications),UMTS (universal mobile telecommunication system), CDMA (code divisionmultiple access, including IS-95, IS-2000, and WCDMA), or LTS (long termevolution)), or other IEEE communication standard.

Definitions

The terms “a” and “an”, when modifying a noun, do not imply that onlyone of the noun exists. For example, a statement that “an apple ishanging from a branch”: (i) does not imply that only one apple ishanging from the branch; (ii) is true if one apple is hanging from thebranch; and (iii) is true if multiple apples are hanging from thebranch.

To compute “based on” specified data means to perform a computation thattakes the specified data as an input.

Here are some non-limiting examples of a “camera”: (a) a digital camera;(b) a digital grayscale camera; (c) a digital color camera; (d) a videocamera; (e) a light sensor or image sensor, (f) a set or array of lightsensors or image sensors; (g) an imaging system; (h) a light fieldcamera or plenoptic camera; (i) a time-of-flight camera; and (j) a depthcamera. A camera includes any computers or circuits that process datacaptured by the camera.

To say that a computer “causes” a device to do an action means that thecomputer outputs one or more signals that directly or indirectly controlthe device and that thereby cause the device to do the action.

The term “comprise” (and grammatical variations thereof) shall beconstrued as if followed by “without limitation”. If A comprises B, thenA includes B and may include other things.

The term “computer” includes any computational device that performslogical and arithmetic operations. For example, in some cases, a“computer” comprises an electronic computational device, such as anintegrated circuit, a microprocessor, a mobile computing device, alaptop computer, a tablet computer, a personal computer, or a mainframecomputer. In some cases, a “computer” comprises: (a) a centralprocessing unit, (b) an ALU (arithmetic logic unit), (c) a memory unit,and (d) a control unit that controls actions of other components of thecomputer so that encoded steps of a program are executed in a sequence.In some cases, a “computer” also includes peripheral units including anauxiliary memory storage device (e.g., a disk drive or flash memory), orincludes signal processing circuitry. The term “computer” also includesany analog computer. However, a human is not a “computer”, as that termis used herein.

As used herein, to say that a scene point “corresponds” to a pixel meansthat light which reflects from the scene point is focused at the pixel.As used herein, to say that a period of time “corresponds” to a complexnumber means that the complex number is calculated based on measurementstaken during the period of time. As used herein, to say that a frequency“corresponds” to a complex number means that the complex number iscalculated based on measurements taken at that frequency.

“Defined Term” means a term or phrase that is set forth in quotationmarks in this Definitions section.

Unless the context clearly indicates otherwise, the terms “depth”,“scene depth” and “distance” each mean distance between a ToF sensor anda point in a scene that is being imaged by the ToF sensor.

For an event to occur “during” a time period, it is not necessary thatthe event occur throughout the entire time period. For example, an eventthat occurs during only a portion of a given time period occurs “during”the given time period.

The term “e.g.” means for example.

Each equation above is referred to herein by the equation number setforth to the right of the equation. For example: “Equation 6” meansEquation 6 above; “Equation 7” means Equation 7 above; and “Equation 13”means Equation 13 above. Non-limiting examples of an “equation”, as thatterm is used herein, include: (a) an equation that states an equality;(b) an inequation that states an inequality (e.g., that a first item isgreater than or less than a second item); (c) a mathematical statementof proportionality or inverse proportionality; and (d) a system ofequations.

The fact that an “example” or multiple examples of something are givendoes not imply that they are the only instances of that thing. Anexample (or a group of examples) is merely a non-exhaustive andnon-limiting illustration.

Unless the context clearly indicates otherwise: (1) a phrase thatincludes “a first” thing and “a second” thing does not imply an order ofthe two things (or that there are only two of the things); and (2) sucha phrase is simply a way of identifying the two things, respectively, sothat they each may be referred to later with specificity (e.g., byreferring to “the first” thing and “the second” thing later). Forexample, unless the context clearly indicates otherwise, if an equationhas a first term and a second term, then the equation may (or may not)have more than two terms, and the first term may occur before or afterthe second term in the equation. A phrase that includes a “third” thing,a “fourth” thing and so on shall be construed in like manner.

“FLI” means fluorescence lifetime imaging.

“For instance” means for example.

Unless the context clearly indicates otherwise, “frequency” meansfrequency of amplitude modulation of light.

To say a “given” X is simply a way of identifying the X, such that the Xmay be referred to later with specificity. To say a “given” X does notcreate any implication regarding X. For example, to say a “given” X doesnot create any implication that X is a gift, assumption, or known fact.

“Herein” means in this document, including text, specification, claims,abstract, and drawings.

As used herein: (1) “implementation” means an implementation of thisinvention; (2) “embodiment” means an embodiment of this invention; (3)“case” means an implementation of this invention; and (4) “use scenario”means a use scenario of this invention.

To say that a calculation is “according to” a first equation means thatthe calculation includes (a) solving the first equation; or (b) solvinga second equation, where the second equation is derived from the firstequation. Non-limiting examples of “solving” an equation include solvingthe equation in closed form or by numerical approximation or byoptimization.

The term “include” (and grammatical variations thereof) shall beconstrued as if followed by “without limitation”.

“Intensity” means any measure of intensity, energy or power. Forexample, the “intensity” of light includes any of the followingmeasures: irradiance, spectral irradiance, radiant energy, radiant flux,spectral power, radiant intensity, spectral intensity, radiance,spectral radiance, radiant exitance, radiant emittance, spectral radiantexitance, spectral radiant emittance, radiosity, radiant exposure,radiant energy density, luminance or luminous intensity.

“I/O device” means an input/output device. Non-limiting examples of anI/O device include a touch screen, other electronic display screen,keyboard, mouse, microphone, handheld electronic game controller,digital stylus, display screen, speaker, or projector for projecting avisual display.

A non-limiting example (in the context of an algorithm) of X not beingnot “known” in advance, is that X is not an input to the algorithm. Forpurposes of the preceding sentence, a value that is computed by thealgorithm is not an “input” to the algorithm, even if the algorithm,after computing the value, uses the value in a subsequent step of thealgorithm.

Unless the context clearly indicates otherwise, the term “lifetime”means “fluorescence lifetime”. A non-limiting example of measuringfluorescence lifetime is measuring average fluorescence lifetime.

“Light” means electromagnetic radiation of any frequency. For example,“light” includes, among other things, visible light and infrared light.Likewise, any term that directly or indirectly relates to light (e.g.,“imaging”) shall be construed broadly as applying to electromagneticradiation of any frequency.

“Listed Patents” means U.S. Pat. No. 6,777,659; U.S. Pat. No. 6,825,455;U.S. Pat. No. 7,012,738; U.S. Pat. No. 8,294,882; U.S. Pat. No.6,323,942; U.S. Pat. No. 6,515,740; U.S. Pat. No. 6,587,186; U.S. Pat.No. 7,405,812; U.S. Pat. No. 7,719,662; U.S. Pat. No. 8,194,233; U.S.Pat. No. 8,314,924; U.S. Pat. No. 8,953,149; U.S. Pat. No. 6,678,039;U.S. Pat. No. 7,060,957; U.S. Pat. No. 7,636,150; U.S. Pat. No.8,786,678; U.S. Pat. No. 7,361,883; U.S. Pat. No. 7,408,627; U.S. Pat.No. 9,516,244; U.S. Pat. No. 7,671,391; U.S. Pat. No. 7,262,402; andU.S. Pat. No. 8,355,117. A “Listed Patent” means one of the ListedPatents. The entire disclosure of each of the Listed Patent is herebyincorporated by reference herein. The entire disclosure of a ListedPatent includes the specification and drawings of the Listed Patent, butdoes not include the patent claims of the Listed Patent.

As used herein, (i) a single scalar is not a “matrix”, and (ii) one ormore entries, all of which are zero (i.e., a so-called null matrix), isnot a “matrix”.

To “multiply” includes to multiply by an inverse. Thus, to “multiply”includes to divide.

The term “or” is inclusive, not exclusive. For example, A or B is trueif A is true, or B is true, or both A or B are true. Also, for example,a calculation of A or B means a calculation of A, or a calculation of B,or a calculation of A and B.

A parenthesis is simply to make text easier to read, by indicating agrouping of words. A parenthesis does not mean that the parentheticalmaterial is optional or may be ignored.

The “phase of the complex numbers” means the phase of each of thecomplex numbers, respectively.

“Scene point” means a location in a scene. As used herein, fluorescencelifetime “of” a scene point means lifetime of fluorescence that occursat the scene point.

As used herein, the term “set” does not include a group with noelements. Mentioning a first set and a second set does not, in and ofitself, create any implication regarding whether or not the first andsecond sets overlap (that is, intersect).

Unless the context clearly indicates otherwise, “some” means one ormore.

As used herein, a “subset” of a set consists of less than all of theelements of the set.

The term “such as” means for example.

As used herein, the term “sweep” of frequencies does not imply any orderof frequencies or direction of change in frequency. Here are somenon-limiting examples: (a) a “sweep” may be from highest to lowestfrequency, or from lowest to highest frequency; (b) frequencies in a“sweep” may be monotonically increasing, monotonically decreasing,monotonically weakly decreasing, monotonically weakly increasing or noneof the above; (c) a given frequency may repeat more than once in asweep, either consecutively or separated by one or more otherfrequencies; and (d) in a sweep of frequencies which includes a highestfrequency, a middle frequency and a lowest frequency, these frequenciesmay occur in any order, such as middle, then highest, then lowest.

To say that a machine-readable medium is “transitory” means that themedium is a transitory signal, such as an electromagnetic wave.

A non-limiting example of a “vector” is a one-dimensional array.(Without affecting this definition in any way, we note that this meaningis common in computer science).

A matrix may be indicated by a bold capital letter (e.g., D). A vectormay be indicated by a bold lower case letter (e.g., a). However, theabsence of these indicators does not indicate that something is not amatrix or not a vector.

Unless the context clearly indicates otherwise, a math expression thatsymbolizes a sequence of numbers in the form n=a, . . . , Y means asequence of integers that starts with integer a and ends with integer Y,where each element of the sequence (except the first element of thesequence) is equal to the preceding element in the sequence plus 1. Forexample: (a) n=0, . . . , 2 means n=0, 1, 2; (b) x=0, . . . , 3 meansx=0, 1, 2, 3; and (c) g=0, . . . , 5 means g=0, 1, 2, 3, 4, 5.

Except to the extent that the context clearly requires otherwise, ifsteps in a method are described herein, then the method includesvariations in which: (1) steps in the method occur in any order orsequence, including any order or sequence different than that described;(2) any step or steps in the method occurs more than once; (3) any twosteps occur the same number of times or a different number of timesduring the method; (4) any combination of steps in the method is done inparallel or serially; (5) any step in the method is performediteratively; (6) a given step in the method is applied to the same thingeach time that the given step occurs or is applied to different thingseach time that the given step occurs; (7) one or more steps occursimultaneously, or (8) the method includes other steps, in addition tothe steps described herein.

This Definitions section shall, in all cases, control over and overrideany other definition of the Defined Terms. The Applicant or Applicantsare acting as his, her, its or their own lexicographer with respect tothe Defined Terms. For example, the definitions of Defined Terms setforth in this Definitions section override common usage or any externaldictionary. If a given term is explicitly or implicitly defined in thisdocument, then that definition shall be controlling, and shall overrideany definition of the given term arising from any source (e.g., adictionary or common usage) that is external to this document. If thisdocument provides clarification regarding the meaning of a particularterm, then that clarification shall, to the extent applicable, overrideany definition of the given term arising from any source (e.g., adictionary or common usage) that is external to this document. To theextent that any term or phrase is defined or clarified herein, suchdefinition or clarification applies to any grammatical variation of suchterm or phrase, taking into account the difference in grammatical form.For example, the grammatical variations include noun, verb, participle,adjective, and possessive forms, and different declensions, anddifferent tenses.

Variations

This invention may be implemented in many different ways. Here are somenon-limiting examples:

In some implementations, this invention is a method comprising: (a) alight source illuminating a scene with a signal of pulsed light that ispulsed non-periodically; (b) a time-of-flight sensor taking measurementsof light incident on the sensor, which incident light includes pulsedlight that has reflected from the scene and includes fluorescent lightthat has been emitted by fluorescent material in the scene in responseto the pulsed light; and (c) one or more computers performingcalculations that include, for each given pixel in a set of pixels inthe time-of-flight sensor (i) calculating a vector of complex numbers byperforming a Discrete Fourier Transform on measurements taken by thegiven pixel, such that each of the complex numbers, respectively, (A)encodes phase and amplitude of light incident at the given pixel, and(B) is calculated based on measurements taken during a temporal portionof the signal, which temporal portion does not correspond to any othercomplex number in the vector, and (ii) calculating, based on phase ofthe complex numbers, fluorescence lifetime of a scene point thatcorresponds to the given pixel. In some cases, the calculating offluorescence lifetime, based on phase of the complex numbers, comprisessolving a non-linear least squares problem. In some cases, thecalculating of fluorescence lifetime, based on phase of the complexnumbers, comprises solving a non-linear least squares problem accordingto Equation 13. In some cases, the pulsed signal encodes a maximumlength sequence. In some cases, the maximum length sequence comprises a31-bit sequence, which 31-bit sequence is0101110110001111100110100100001. In some cases, the pulsed signalencodes a Gold sequence. In some cases, the calculations for the givenpixel include calculating, based on phase of the complex numbers, scenedepth of a scene point that corresponds to the given pixel. In somecases, for the given pixel, the one or more computers calculate, basedon phase of the complex numbers, both depth of the scene point andlifetime fluorescence of the scene point by solving the non-linear leastsquares problem. In some cases, for the given pixel, the one or morecomputers calculate, based on phase of the complex numbers, both depthof the scene point and lifetime fluorescence of the scene point bysolving the non-linear least squares problem according to Equation 13.Each of the cases described above in this paragraph is an example of themethod described in the first sentence of this paragraph, and is also anexample of an embodiment of this invention that may be combined withother embodiments of this invention.

In some implementations, this invention is a system comprising: (a) alight source; (b) a lock-in time-of-flight sensor; and (c) one or morecomputers that are programmed (i) to cause the light source toilluminate a scene with a signal of pulsed light that is pulsednon-periodically, (ii) to cause the time-of-flight sensor to takemeasurements of light incident on the sensor, which incident lightincludes pulsed light that has reflected from the scene and includesfluorescent light that has been emitted by fluorescent material in thescene in response to the pulsed light, and (iii) to perform calculationsthat include, for each given pixel in a set of pixels in thetime-of-flight sensor (A) calculating a vector of complex numbers byperforming a Discrete Fourier Transform on measurements taken by thegiven pixel, such that each of the complex numbers, respectively, (I)encodes phase and amplitude of light incident at the given pixel, and(II) is calculated based on measurements taken during a temporal portionof the signal, which temporal portion does not correspond to any othercomplex number in the vector, and (B) calculating, based on phase of thecomplex numbers, fluorescence lifetime of a scene point that correspondsto the given pixel. In some cases, the calculating of fluorescencelifetime, based on phase of the complex numbers, comprises solving anon-linear least squares problem. In some cases, the calculating offluorescence lifetime, based on phase of the complex numbers, comprisessolving a non-linear least squares problem according to Equation 13. Insome cases, the calculations for the given pixel include calculating,based on phase of the complex numbers, scene depth of a scene point thatcorresponds to the given pixel. In some cases, for the given pixel, theone or more computers calculate, based on phase of the complex numbers,both depth of the scene point and lifetime fluorescence of the scenepoint by solving the non-linear least squares problem. In some cases,for the given pixel, the one or more computers calculate, based on phaseof the complex numbers, both depth of the scene point and lifetimefluorescence of the scene point by solving the non-linear least squaresproblem according to Equation 13. In some cases, the pulsed signalencodes a maximum length sequence. In some cases, the maximum lengthsequence comprises a 31-bit sequence, which 31-bit sequence is0101110110001111100110100100001. In some cases, the pulsed signalencodes a Gold sequence. In some cases, the light source comprises oneor more laser diodes. In some cases, the system further comprises adielectric filter configured to filter out pulsed light that reflectsfrom the scene. Each of the cases described above in this paragraph isan example of the system described in the first sentence of thisparagraph, and is also an example of an embodiment of this inventionthat may be combined with other embodiments of this invention.

Each description above of any method or apparatus of this inventiondescribes a non-limiting example of this invention. This invention isnot limited to those examples, and may be implemented in other ways.

Each description above of any implementation, embodiment or case of thisinvention (or any use scenario for this invention) describes anon-limiting example of this invention. This invention is not limited tothose examples, and may be implemented in other ways.

Each Figure that illustrates any feature of this invention shows anon-limiting example of this invention. This invention is not limited tothose examples, and may be implemented in other ways.

The Provisional Application does not limit the scope of this invention.The Provisional Application describes non-limiting examples of thisinvention, which examples are in addition to—and not in limitationof—the implementations of this invention that are described in the mainpart of this document. For example, if any given feature described inthe Provisional Application is different from, or in addition to, thefeatures described in the main part of this document, this additional ordifferent feature of the Provisional Application does not limit anyimplementation of this invention described in the main part of thisdocument, but instead merely describes another example of thisinvention. As used herein, the “main part of this document” means thisentire document (including any drawings listed in the Brief Descriptionof Drawings above and any software file listed in the Computer ProgramListing section above), except that the “main part of this document”does not include any document that is incorporated by reference herein.

Any document that is incorporated by reference herein (“incorporateddocument”) does not limit the scope of this invention (including thescope of any hardware, hardware component, method, process, step,software, algorithm, feature, or technology that is described in themain part of this document). For example, if any given feature describedin any incorporated document is different from, or in addition to, thefeatures described in the main part of this document, this additional ordifferent feature of the incorporated document does not limit anyimplementation of this invention described in the main part of thisdocument.

The above description (including without limitation any attacheddrawings and figures) describes illustrative implementations of theinvention. However, the invention may be implemented in other ways. Themethods and apparatus which are described herein are merely illustrativeapplications of the principles of the invention. Other arrangements,methods, modifications, and substitutions by one of ordinary skill inthe art are therefore also within the scope of the present invention.Numerous modifications may be made by those skilled in the art withoutdeparting from the scope of the invention. Also, this invention includeswithout limitation each combination and permutation of one or more ofthe implementations (including hardware, hardware components, methods,processes, steps, software, algorithms, features, or technology) thatare described or incorporated by reference herein.

What is claimed is:
 1. A method comprising: (a) a light sourceilluminating a scene with a signal of pulsed light that is pulsednon-periodically; (b) a time-of-flight sensor taking measurements oflight incident on the sensor, which incident light includes pulsed lightthat has reflected from the scene and includes fluorescent light thathas been emitted by fluorescent material in the scene in response to thepulsed light; and (c) one or more computers performing calculations thatinclude, for each given pixel in a set of pixels in the time-of-flightsensor (i) calculating a vector of complex numbers by performing aDiscrete Fourier Transform on measurements taken by the given pixel,such that each of the complex numbers, respectively, (A) encodes phaseand amplitude of light incident at the given pixel, and (B) iscalculated based on measurements taken during a temporal portion of thesignal, which temporal portion does not correspond to any other complexnumber in the vector, and (ii) calculating, based on phase of thecomplex numbers, fluorescence lifetime of a scene point that correspondsto the given pixel.
 2. The method of claim 1, wherein the calculating offluorescence lifetime, based on phase of the complex numbers, comprisessolving a non-linear least squares problem.
 3. The method of claim 2,wherein, for the given pixel, the one or more computers calculate, basedon phase of the complex numbers, both depth of the scene point andlifetime fluorescence of the scene point by solving the non-linear leastsquares problem.
 4. The method of claim 1, wherein the calculating offluorescence lifetime, based on phase of the complex numbers, comprisessolving a non-linear least squares problem according to Equation
 13. 5.The method of claim 4, wherein, for the given pixel, the one or morecomputers calculate, based on phase of the complex numbers, both depthof the scene point and lifetime fluorescence of the scene point bysolving the non-linear least squares problem according to Equation 13.6. The method of claim 1, wherein the pulsed signal encodes a maximumlength sequence.
 7. The method of claim 6, wherein the maximum lengthsequence comprises a 31-bit sequence, which 31-bit sequence is0101110110001111100110100100001.
 8. The method of claim 1, wherein thepulsed signal encodes a Gold sequence.
 9. The method of claim 1, whereinthe calculations for the given pixel include calculating, based on phaseof the complex numbers, scene depth of a scene point that corresponds tothe given pixel.
 10. A system comprising: (a) a light source; (b) alock-in time-of-flight sensor; and (c) one or more computers that areprogrammed (i) to cause the light source to illuminate a scene with asignal of pulsed light that is pulsed non-periodically, (ii) to causethe time-of-flight sensor to take measurements of light incident on thesensor, which incident light includes pulsed light that has reflectedfrom the scene and includes fluorescent light that has been emitted byfluorescent material in the scene in response to the pulsed light, and(iii) to perform calculations that include, for each given pixel in aset of pixels in the time-of-flight sensor (A) calculating a vector ofcomplex numbers by performing a Discrete Fourier Transform onmeasurements taken by the given pixel, such that each of the complexnumbers, respectively, (I) encodes phase and amplitude of light incidentat the given pixel, and (II) is calculated based on measurements takenduring a temporal portion of the signal, which temporal portion does notcorrespond to any other complex number in the vector, and (B)calculating, based on phase of the complex numbers, fluorescencelifetime of a scene point that corresponds to the given pixel.
 11. Thesystem of claim 10, wherein the calculating of fluorescence lifetime,based on phase of the complex numbers, comprises solving a non-linearleast squares problem.
 12. The system of claim 11, wherein, for thegiven pixel, the one or more computers calculate, based on phase of thecomplex numbers, both depth of the scene point and lifetime fluorescenceof the scene point by solving the non-linear least squares problem. 13.The system of claim 11, wherein the maximum length sequence comprises a31-bit sequence, which 31-bit sequence is0101110110001111100110100100001.
 14. The system of claim 10, wherein thecalculating of fluorescence lifetime, based on phase of the complexnumbers, comprises solving a non-linear least squares problem accordingto Equation
 13. 15. The system of claim 14, wherein, for the givenpixel, the one or more computers calculate, based on phase of thecomplex numbers, both depth of the scene point and lifetime fluorescenceof the scene point by solving the non-linear least squares problemaccording to Equation
 13. 16. The system of claim 10, wherein thecalculations for the given pixel include calculating, based on phase ofthe complex numbers, scene depth of a scene point that corresponds tothe given pixel.
 17. The system of claim 10, wherein the pulsed signalencodes a maximum length sequence.
 18. The system of claim 10, whereinthe pulsed signal encodes a Gold sequence.
 19. The system of claim 10,wherein the light source comprises one or more laser diodes.
 20. Thesystem of claim 10, further comprising a dielectric filter configured tofilter out pulsed light that reflects from the scene.